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ABSTRACT 

The unidimens^ionality of the reading and mathematics 
tests.^of the California Assessment Program was invest igat^ed. During 
April ofi 1980, all third ^grade , students in California took a te^t 
consisting of reading, mathematics, and written^ language items. Item 
intercoxrelation matrices were constructed for each skill, and the 
latent roots extracted. Methods of outlier analysis were applied to 
look for lai^ge drops in the siz£ of the latent roots from each 
matrix,. Mathepatici, skills 4:ended' to, have smaller ^initial roots and a 
second root cl^d^ser to the first than the-^reading skills. The evidence 
does not support the hypothesis of unidi^nsionality for the mon^y 
and f rations skills, but does not contradict the hypothesis for the 
mathematical applications skill or for reading skills. (Author/BW) 
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AUTHOR *S NOTE 



.This article was originally written for presentation cit the 
annual meeting of the American Educational* Research. Associa- 
tion,^ New York> March/- •'ISSS, Special |:hanks are ov^ed to 
I^§le Q§rlfeoi^)^f tte 'California* Ddpaftment oi /EduQation. ' Dr^ 

,C^rJ;©on'£?«' iBl\couragemeivt> • criticism', and generosity are 



* s 



largely ,resppnsiblQ fot_.any success- -of -this, study, I am 

J , • , • / . ' • ' - ' . ' 

es^^edially, indebted to Dave Wiley and Annegret Harnischf eger 

for th^ir exceptional ly lucid discussion of the problem of, 
undim^nsionality of, CAP skills arid the ^pi peneti*ating ana- 
'iyses of the phenomenon, I^'of course, retain sole •respon- 
sible for any remainiivg errors * of logic on-cxecution, ^ . The 

views expressed here are not necessarily those of the Cali- 

» »» ' 

fornia Departm^rvt of Education. 



■ABSTRACT 

This paper reports the results of ♦an. iiw.estigation- of the 
unidim.ej^si'onality of the reading and mathematics" tests ^df 

the Califcwrnia ' Assessment Program. ' ftem int^efcoTr elation 

' . - - . ' ' ''^ "^-"■^'fe ' ' ; 

matrices were ^on^i>ruc;ted f or eacJf^^^^ni* .c^nd". the latent 

roots ex^traoited, fl^thpds'of putlior analysis' are applied to 

look- for large (Jrpps in .the size of the latent' roots from- 

* * / • * " \ 

each ma\rix, Results of factor analyses are examined, and 

item content is examined to check the outcomes of the sta- 
tistical procedures. 



INTRODUCTION 

In- 1980 the, California Assessment Program^ (CAP) .intro- 
duced a new test for the third grade> which is administered 
• « . ^ 

each spring to more, than Z80,000 students'. Scaled scores 
are f eporT:edr-fo\ 60 different skills in reading, written 
language, and mathematics, *•* The**model used -to estimate the 
scores is based on item response curve (PRC) theory, and 
'includes parameters for the' difficulty and reliability of 
items (Mislevy and Bock, 1980), > 

^ 

The mathejnatical model used to measure .performance on a 
CAP skill, ' assumes that the probability of success on an 

^it;3m depends only on the item parameters and on examinee 
ability. This is the assumption of unidimensionality. Lord^ 
(1980, pp. ' ZO-Zl) suggests that if the items measure lone 
dimension, and if the IRC model holds, . then 'the matrix of 
tetrachoric intercorrelations will be of unit rank. To 
examine this possibility Lord suggests plotting the latent 
roots of the correlation matrix. "If (1) the first root ia^ 
large convpared to the second and (E) the second root is not 
much larger tharv ^ny^ of the othei^, then the items are 
approximately unidimensional, '^A direct applica-Lion of €his' 

^procedure ' to CAP's third grade 't*eM is not possiJile for 
several reasons. The CAP test has 30 forms and is adminis-' 



tered according to a multjple matrix sampling design. 
Although sicills contain f rom' -^en to twenty items, feach stu- 
dent sees at most one of these. Furthermore, the school is 
the basic unit for reporting and analysis. Mislevy and Bod; 
(1980) describe* the application of IRC theory to this situa- 
tioti. \ They note that- "under the CAP multiple matrix sam- 
plii^ design, each of a school's responses to items from a 
giv^n skill has been obtained from • a different randomly 
selected pupil. ' For the school level analysis, then,' the 
'standard assumption in IRC theory of 'Iqcal* or 
•conditional* independence is satisfied perfectly." Misle- 
vey and Bock's paper contain^ a detailed. , ^ctjount o£ the' 
derivation of scores. 



An analysis of unidimensionality of CAP skills was under- 
taken by Wiley and Harnischf eger (1981) using data and ana- 
lyses supplied by the author* Their procedure involved 
examination of the ^iistribution of successive differences of 
logarithms lof .roots to locate differences in facing of 
roots 'which "might indicate discrepancies in rot>t size. 
. ^ ' '.Method 

Materials and procedure? The third grade- test^ wfis admin- 
istefed/during Aptil of J1980 to all third grade students 
according to standardized procedures hy school personnel. 
Each student, was asked -.to comple-|;e one. x>f thirty different 
forms, spiraled to insure even distribution of, all forpis. 



eaph form containing .nine reading items, twelve mathematics, 
*gnd thirteen written* language items.' ^ Allocation- of reading 
.►and mathematics items to skills is ' shown in Tables 1 and 2. 
Ample time was allowed for completion," Aftdr test ^adminis- 
tration the documents wore^'collected and processed off -site. 



Insert tables 1 *and 2 about hefe. 



Procedures^ for. development of the items were designed to 
assure content validity.' Spercif icatibns • for eafch'sk^ll were 
faref uJLiyJiiand narrowly defined. Items were written, in 
accordanc&V'^ith the specifications and miderwent ^ several 
rounds of content review and field testing! Given the rela-^ 
ti've fineness 6i definition of specifications, these proce- 



dures'^tended to^ support tlie hypothesis of undiminsionality 

of item content in each skill. This is a matter of consid- 

ererd. logical, ^ and nQt statistical judgement. Vjiil® statis- 
ts. 

tioal procedlires discussed here provide a useful adjunct for 
making a judgement of unidimensionality, ^ they are not pro- 
posed as a definitive criterion. ^ ' ■ 

Analysis, For ' each item in a ^^^-^-i number of 



attempts and the number of correct responses were tabulated 
and used .to calculate a iogit item score for each ^^hool. 
Only larger scho.ols, havin'g at least two responses'^er item 
^f^ere included in the analysis. Latent roots v?ere entratM^ed 



from, the' intercprrelation* matrix of the logit ite\n scores. 
The matjrix wa» factor analyzed by the principal axis\method> 
factors with eigenvalues greater than* 1.0 w^re retaine(J, and 
varimaK rotated. This .procedure was repeated, ior .each of 
the 17 reading skills and* the 20 mathematics skills. 

Statistical procedures described by Barnett and Lewis 
(1978) used to examine distributions for outliers can be 
app^lied to examine the hypothesis of unid'imensionality. 
Lord'^s procedure asks whether the second root, in addition 
to the first* is, large with respect to remaining roots. 
This is likfs^ asking whether the second root, in'addition to 
tlie f ir^tf i^; an upper outlier with respect to remaining 
roots. If the answer is yes> this can be interpreted as 
lack of support for the hypothesis of unidimensionali'ty. 

A class of statistics disG^ussed by Barnett dnd Lewis 

involve tsrking ratios of difference scores. . The numerator 

of the ratio is the difference between the outlier and some 

function of the remaining observations. , The denominator is 

a measure of spread* such as a range or standard deviation. 

The search for n^re than one upper outlier may involve con- 
t 

secutive or sequential testing. ^Each value is compared with 
its next lower, neighbor or with a linear combination of 
lower neighbors. 



Results . \ 

Table 3 displays for e^ch skill the first .two roots^ the 

ir « 

mean of the remaining roots and .the ratio 

(ropt2-mean)/(rdotl-rootZ) » Relatively larger ratios can be 

interpreted in terms of a^^econd root that is closer to the 

ft 

first root than.'to the mass of remaining roots. In .generad> 
math skills tend to have larger ratioa and smaller initial 
rooti^than reading skills. The largest ratio belongs to the 
nature of numbers-money and fractions skill. 



Insert' table 3 about here. 

Figure l' is a scatterplot with (rootl-root2)/range. and^ 

Jroot2-mean)/range on the horizontal axis. Reading skills 

•are ^denoted by ' "R"/ and mathematics skills by "M". Using 
• - ft * 

the range as a divisor tended to ^produce a straight line 
plot> as shown, while similar plots using no divisor or the 
standard devi^ation were more amorphous, e.g. figure 2. . The 
reading skills are grouped in the upper left corner of the 
plot. This indicates that in reading there was a larger 
drop from the first to the second root than in math, and 
that the ^second root in reading tends to be closer to the 
mean of remaining roots. The two extrehie points in th0 
-lower right of the plot represent money and fractions and 
basic skills operations. 



Insert figures 1, Z, 3 and 4 about here. 

Figure 3 is a plot of the^ roots of the money and frac- 
tions skill. . For eonip9rison purposes figure 4^ is a plot of 
, the roots of the nature of numbers applications skill. The 

♦ 4 

second plot corresponds more nearly to the ideal for a uni- 
dimensional skill. "Plots of the logs of the eigenvalues for 
these two skills are |;hown in iigurfes. 5 and 6, The first 
root is off the scale on both plots. * The roots for. the. 
applications skill appea^ to form- an jinbrokeii curve* while 
,the roots for *money afi'd fractions do not; , with t^e second 
and possibly the third root seeming' to* deviate, ' 



-I 



^ JInsert figures 5, 6* 7 and 8 about here. 



A plot of the linear regre^i^on of the logs^ of the' roots 
on serial position, excluding thb first root, is shown in 
Figure' 5. The equation of t^e line is y(predicted) = 
-.0S4^x + .347 . ,With the exception of the s&ttohd root, 
most of the points lie close to the line. 



deviation i& .137, and the the standard' error of estimate is 
.055, this point is 2.5 stand^ird errors. from the line. 

Figures. 7 and 8 display consecutive differences of logs 
of roots: That is, the first value is (logl-logZ), the sec- 



ojid is <4og2 - log3)> and • -so on, A^ain> for raoney sjnd 

fractibris the second root does nqt^ appear to' b'elong to the 

• * * • » # 

remaining mass. By*c6ntrast> the ^applications sk\ll-has one 
•large initial differeuce^ followed by a sequenpe of much, 
smaller differences having no apparent upward or^downwar.d 
trend. * , 

For reference with the factor analysis, the items in the 

/ * 

money aod - fractions skill are displayed in^the appendix. 

Based on logical, analysis of content, there app&'ar to .be two 

distinct types: Those involving the shading of a geometric 

* • » 

fjgure; and those relating to ^r^oney. • There are several 

major variations in format./-. Seven money itejns involve 

tran^^ting a picture of coins into, an amount of money, and 

twd involve translating numc^ric to- lo^gha'nd written ^amounts. 

Three fractions items' involve translarting a numeric fraction 

into €? -shaded geometric shajDe and the remaining involve 

translating a shaded shape into a particular numeric frac- 

taon. 

« - 

Results of the factor analysis are shown in Table 4. 

Although not^totally unambjLguous, some patterns may:-^ dis- 

■ . ' • . * 

cerned. Items Z, 3, 4 and 5 (fractions) load positively .on 
■^Tfcictor one. Items 8 and 10,.- and to a lesser extent 12 and 
14' (money), loaded positively on factdp two. ^ Items 5, 11 
^ and 13/(monlBy) loaded negatively on factor three.' The- situ- 
ation is not as clear for^ factors fout and five. Items ^4 



11 



• • 8 

^fractions) attd 12 and 13 (money) . load on ia<itoi;^ four> and 
items 1 and 15 (fractions) and 7 and 11 Cmonpy) load Iposi- 
tively on factor five. * * . ; - 



Jnsert table 4^ about here. 



A question remains as' to the extent item formats might 
-hcive influenced the * factor loadings. One of the fraction 
item formats includes a fraction in the stem and options are 
geometr^ical figures (items I, 3, and 15^- Thg other frac- 
tion item format includes a gqomQ,trical figure ,in, ths stem* 

' • -.^ ^. ' . . \ ' ' 

and the ^opti^ns are fractions ( itws 2/4 an^} 5). In fact, ^ 
items 2/ 4 amd 6 load on factor one, and items 1 and 15 load 

on^factc^r .5. UnexpecteJilyv dtenj ^ loads on fcictor one\ 

. • ' ' . ^ ' ' \ 

'While the evidence for a lormat effegt is suggestive, it ^ . 
* • \ • * 

-certainly is not <conclusiven-- A similar result obtained with 
* * ♦ » > ' - # 

money itemOf o/^mats. 'Items 9 and 11, involving translation" 

of of numerifcal and written amouivtsi, did load together on 

V , If ' • * 

factor three. Unexpectedly;^^ they are \ joined by ^item 5,* 

*' ' ' 

which involved recognition of a coin denomination. Addi- 

t&nlally, item • 11 loaded rather heavily on factor five, 
. * f «..-^ 

' * •* ' . ' 

which seems to contain a little of everything. 

Disc\ission •* ^ * . ' , 

■. . . • ■ \ 

' . One of the stVikir^J features of Table 3 is the diftex*encfe / 
between mathematics and reading skills. . Reading skills tend 



to have larger initial roots, smaller ratio^/ aVid have a 

smaller' range^ of ratios. By contrast the mathematicsv ratios 

have, a m^ich grea-ter range, with basic facts operations and 

money and fracti%^l? as two upper values. If the ratio by 

itself could be' considered an index of unidimensionality, it 

seems that reading - skills are more so than mathematics 

: skills. Even so, it is tcue that great care wasi lavislied 

on skills to assure narrowly defined content. Possibly var- 

* 

iations in mathematics item formats, . which have no corre-- 
spending counterparts in reading, are, partly responsible for 
the "difference. ^ . ^^^3 

•The plots 'of difference statistics, figures 1 and 2, show 
a clumping and segregation of reading skills, confirming a 
similar trend in Table 3 o*f roots and ratio^. Again, . the 
two math' skills which stand out from thQ :rest are basic 
facts operat'ions and money and fractions. . 

Figure 1, with its> nearly linear plot 3tat>ds in contrast, 
to the . more amorphous shape of figure^ 2. "^^^^V 
dividing by the range increases the value ^of of-Ahe variable 
on the ordinate, (rootl-root2)/r'ange, ■ relative to the value 
on the abscissJ, (root2-mean)>r.ange. For ^skills with rela- 
tively large second roots the values on^^the ordinate are 
small relative to t"hose on the abscissa. As a^result,' the 
plot displays such outlying skills, on the lower .right* 



V • . ^ . 10 
.... , » 

Figure 3 is a plot ^of the latent roots" for money and 

• fractions^ and figure |^e4^^|L«^ys roots for the applica- 
^ tions^skill. The relatively larger second root of inon'ey and 

fractions is the main difference between the two plots. The 
. remaining figure^ attempt to make|the difference more evi- 
dent. If the logs of the roots are plotted^ e.g, in figures- 

4 and 5> the descrea^se in value of the root^ with increasing 

♦ 

serial position is more evident^. Tentatively^ one can com- 

• « 

pute a regression line., deviations from the line. On this 
. basis the second root of m^ey and fractions wa$ shown to be 
an outlier. Visual inspection of the applications r-^ts 



indicates no such outliers. Certain limitations of this 
approach nnist be acknowledged. ^It ^ assumed that ^he dis-^ 
tributions of latent roots satisfy those needed for regres- 
sion. Even if tliese assumptions do riot strictly holdr ' it 
may be that the analysis .yields -acceptable rule of thumh# 
and in this case it appears to be so. 

Figures 7 and 8 are* plots of differences' of successive 



pairs 6| eigenvalues for the two skills under consideration. 
Large values indicate a large drop ' in root size. The 



results are generally consistent with earlier figures. The 
applications skill shows a large initial drop» followed by 
much smaller diffenqes. The money and fractions skill shows 
a much smaller initial drop, followed by relatively large 
sqcond and third differeces> with the remaining differences^ 
similar to those in applications.- 



. ' ' • / . . 

11 

♦ • 1 

9 

Comparison of the factor loading;5^ displayed iii Table^^ 
with the items. in the appendix ^eems to- confirm that factor * 
one respresents mpB^ ^;iS^ft'^s"'Snd, that factors two .and three' 
reprfesent fractions. Factors four anf fixe were mixed. 

There was some evidence for format factors. However, this 

^ ' \ 

hypothesis was clouded somewliat by the potpourri, of items 
loading on factors four* and fiv^. 

Several techniques have deen presented for examining the 
hypothesis of unidimensionality. The evidence does .not s\ip- ^ \ 
port this Ii^othesis for the money and fraction^ skills but 
did not coi^tradict the hypothesis for thL applications 
skin.. The result confirmed a feature thajt had. been butlt 

V* 

ii\to the test. The money an^ fractions skill was designed 
to have two types of items. This was the legitimate and 
considered decision of ^the committee responsible for design 
of the test> and related to a peed for a certain type of 
score. ' Making statistical judgements- about unidimensional- 
ity waiS aided considerably by the presence of many other 
skills for comparison purposes. This helped provide a sense 
of what was genuinely unusual an^- what was a mild-aberra-^ 
tion. In addition, there were definite differences across 
content areas. Comparison o'f an isolated" mathematics skill 
with an isolated reading skill. woifld have been misleading. 
In hindsight, it seems important to be aware of such differ- 
ences. ' * 



' ^ 12 • 

♦ J* ... 

EKamination of the figures is probably a reasonable way 
to make judgements about uni dimensionality/ Still, there is 

^ often a felt ^tieed tgr a statistic and a signif ICjEince test. 

. Keeping in h^ind caveats about the assiTmptions of regression 
analysis, it seems that examination of residuals would be 
one way to precede. In final analysis one must take a close 
look ^at item content. ' Here factor analysis proved to be a 
useful adjunct. . , - - • 



1 



8 



Bibliography 



Barnett,^ V. £ Lewis, T. Outliers in Statistical Data . New 
• Yorki; Wiley, 1978. - 

Lord, F.| C Novick, M. Statistical theories of mental test 
scores . Henlo Park/ CajLifornia; Addison WesleV* 1968, ' 



Lord, F. Applications of Item Resppnse Theory v to Practical 
Testing Frdblems Hillsdale, New Jersey; Earlbaum, 1980. 

Mlsrevy, R. and Bock^, D. Estimation ©1 CAP scaled scores, 
, International Educational Services Technical Report # 
100. Chicago; International ^duqational Services, ^ 
(Undated). - • i*: 

Wdley, D. C Harnischf eger. A." Assess tncT»^rfg Reporting CAP 
Results . Kenilworth, Illinois: Th6 Beacon Institute, 
Inc. , 1981. . ^ ^ -^r^ 




- 13 - 



10A5-0403-7 r0501-A es3A6/lca/B2 



> Table | ' .' 

Survey of Basic Skills: Grade 3. ^..-^' 
Reading Items ' ' 



Skill area 

Word Identification' 

Phonics 
Vowels 
Consonants 



Nuwnber of Items 
60. . f 
3Q- 



Structural analysis 

Prefixes, suffixes, and roots 
Contractions and compound words 

Vocabulary 

Recognizing word meanings 
Using context' 

Com&rehensloti 

Literal , 
^ Details ^ 

• - From a 'single sentenca u ^ 
1 - From two or three sentences 



Pronoun references 
Sequence » 

Inferential 
• Main ideas 

C^use and effect 
\ Drawing conclusions^ 

- About characters 
• Prom details 

- Jrom overall 

Stud^ locatlonal 

Alphabetizing, , 
Table of contents 



30 

18 

12 , 

.30" 

. 16 
14 

150. 

?^ 
37 

20 
17 

18 
19 

76 
19 
20 
37 
15 
12 
•10 

30 

' n 

15 
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Table ^ 

'; Survey of Basic Skills; > "tirade 3 
. ^ Mathematic Items ^ .1^ * 

\ 

Skill area * • ' Number of items 



vputiting and place value '45 



Skills • ^ ^ 



30 



Applications r 15 

Operations ' * 255 

Basic facts • 25 

, Addition • ' ' ' 30 

Subtraction ^ 30 

Muitiplicatipn * \ 30 

Applications • . . 40 

Basic facts . ' - 13 

Addition/subtraction ^ 15 

Multiplication ^ 12 

Nature of numbers av^ properties - <i . 45 

' . ' ' - . ' ' 

Properties^ and relationships ' 15 

Money and fractions "15 

, Ji^pplications ' ^ • 15 

.Geometry . » . ^ 30 

Skills ^ • ; • ' 20 

Applications * 10 

Measurement ' . -40 

' Linear measu/es , . ' 15 

0 ther' measures . * . I ' ^ '15 

Applications ^ . 10 

{'Patterns and graphs. . ' ' ' * • * ' 30 

..Skills , ' , " ' 15 

Applications ^ • . . / . 

An^ilysis and models - * .15 
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